Search for: All records

High-quality benchmarks are essential for evaluating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solution as they are prone to data leakage and inflated performance results. To address these challenges, we propose PhantomWiki: a pipeline to generate unique and factually consistent document corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is neither a fixed dataset, nor is it based on any existing data. Instead, a new PhantomWiki instance is generated on demand for each evaluation. We vary the question difficulty and corpus size to disentangle reasoning and retrieval capabilities respectively, and find that PhantomWiki datasets are surprisingly challenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities. 

Developing prompt-based methods with Large Language Models (LLMs) requires making numerous decisions, which give rise to a combinatorial search problem over hyper-parameters. This exhaustive evaluation can be time-consuming and costly. In this paper, we propose an adaptive approach to explore this space. We are exploiting the fact that often only few samples are needed to identify clearly superior or inferior settings, and that many evaluation tests are highly correlated. We lean on multi-armed bandits to sequentially identify the next (method, validation sample)-pair to evaluate and utilize low-rank matrix factorization to fill in missing evaluations. We carefully assess the efficacy of our approach on several competitive benchmark problems and show that it can identify the top-performing method using only 5-15% of the typical resources—resulting in 85-95% LLM cost savings. Our code is available at https://github.com/kilian-group/banditeval. 

Abstract Disease is a key driver of community and ecosystem structure, especially when it strikes foundation species. In the widespread marine foundation species eelgrass (Zostera marina), outbreaks of wasting disease have caused large‐scale meadow collapse in the past, and the causative pathogen,Labyrinthula zosterae, is commonly found in meadows globally. Research to date has mainly focused on abiotic environmental drivers of seagrass wasting disease, but there is strong evidence from other systems that biotic interactions such as herbivory can facilitate plant diseases. How biotic interactions influence seagrass wasting disease in the field is unknown but is potentially important for understanding dynamics of this globally valuable and declining habitat. Here, we investigated links between epifaunal grazers and seagrass wasting disease using a latitudinal field study across 32 eelgrass meadows distributed from southeastern Alaska to southern California. From 2019 to 2021, we conducted annual surveys to assess eelgrass shoot density, morphology, epifauna community, and the prevalence and lesion area of wasting disease infections. We integrated field data with satellite measurements of sea surface temperature and used structural equation modeling to test the magnitude and direction of possible drivers of wasting disease. Our results show that grazing by small invertebrates was associated with a 29% increase in prevalence of wasting disease infections and that both the prevalence and lesion area of disease increased with total epifauna abundances. Furthermore, these relationships differed among taxa; disease levels increased with snail (Lacunaspp.) and idoteid isopod abundances but were not related to abundance of ampithoid amphipods. This field study across 23° of latitude suggests a prominent role for invertebrate consumers in facilitating disease outbreaks with potentially large impacts on coastal seagrass ecosystems. 

Eelgrass creates critical coastal habitats worldwide and fulfills essential ecosystem functions as a foundation seagrass. Climate warming and disease threaten eelgrass, causing mass mortalities and cascading ecological impacts. Subtidal meadows are deeper than intertidal and may also provide refuge from the temperature-sensitive seagrass wasting disease. From cross-boundary surveys of 5761 eelgrass leaves from Alaska to Washington and assisted with a machine-language algorithm, we measured outbreak conditions. Across summers 2017 and 2018, disease prevalence was 16% lower for subtidal than intertidal leaves; in both tidal zones, disease risk was lower for plants in cooler conditions. Even in subtidal meadows, which are more environmentally stable and sheltered from temperature and other stressors common for intertidal eelgrass, we observed high disease levels, with half of the sites exceeding 50% prevalence. Models predicted reduced disease prevalence and severity under cooler conditions, confirming a strong interaction between disease and temperature. At both tidal zones, prevalence was lower in more dense eelgrass meadows, suggesting disease is suppressed in healthy, higher density meadows. These results underscore the value of subtidal eelgrass and meadows in cooler locations as refugia, indicate that cooling can suppress disease, and have implications for eelgrass conservation and management under future climate change scenarios. This article is part of the theme issue ‘Infectious disease ecology and evolution in a changing world’. 

Seagrass meadows provide valuable ecosystem benefits but are at risk from disease. Eelgrass ( Zostera marina ) is a temperate species threatened by seagrass wasting disease (SWD), caused by the protist Labyrinthula zosterae . The pathogen is sensitive to warming ocean temperatures, prompting a need for greater understanding of the impacts on host health under climate change. Previous work demonstrates pathogen cultures grow faster under warmer laboratory conditions and documents positive correlations between warmer ocean temperatures and disease levels in nature. However, the consequences of disease outbreaks on eelgrass growth remain poorly understood. Here, we examined the effect of disease on eelgrass productivity in the field. We coupled in situ shoot marking with high-resolution imagery of eelgrass blades and used an artificial intelligence application to determine disease prevalence and severity from digital images. Comparisons of eelgrass growth and disease metrics showed that SWD impaired eelgrass growth and accumulation of non-structural carbon in the field. Blades with more severe disease had reduced growth rates, indicating that disease severity can limit plant growth. Disease severity and rhizome sugar content were also inversely related, suggesting that disease reduced belowground carbon accumulation. Finally, repeated measurements of diseased blades indicated that lesions can grow faster than healthy tissue in situ . This is the first study to demonstrate the negative impact of wasting disease on eelgrass health in a natural meadow. These results emphasize the importance of considering disease alongside other stressors to better predict the health and functioning of seagrass meadows in the Anthropocene. 

Climate change is one of the greatest challenges facing humanity, and we, as machine learning (ML) experts, may wonder how we can help. Here we describe how ML can be a powerful tool in reducing greenhouse gas emissions and helping society adapt to a changing climate. From smart grids to disaster management, we identify high impact problems where existing gaps can be filled by ML, in collaboration with other fields. Our recommendations encompass exciting research questions as well as promising business opportunities. We call on the ML community to join the global effort against climate change. 

Abstract Effective solutions to conserve biodiversity require accurate community‐ and species‐level information at relevant, actionable scales and across entire species' distributions. However, data and methodological constraints have limited our ability to provide such information in robust ways. Herein we employ a Deep‐Reasoning Network implementation of the Deep Multivariate Probit Model (DMVP‐DRNets), an end‐to‐end deep neural network framework, to exploit large observational and environmental data sets together and estimate landscape‐scale species diversity and composition at continental extents. We present results from a novel year‐round analysis of North American avifauna using data from over nine million eBird checklists and 72 environmental covariates. We highlight the utility of our information by identifying critical areas of high species diversity for a single group of conservation concern, the North American wood warblers, while capturing spatiotemporal variation in species' environmental associations and interspecific interactions. In so doing, we demonstrate the type of accurate, high‐resolution information on biodiversity that deep learning approaches such as DMVP‐DRNets can provide and that is needed to inform ecological research and conservation decision‐making at multiple scales.